This blog post describes our experience debugging a failing test case caused by a cross join query running "too fast." Because the root cause of fail test case spans across multiple layers--from Apache Spark to the JVM JIT compiler-- we wanted to share our analysis in this post. The vast majority of big data SQL or MPP engines follow the Volcano iterator architecture that is inefficient for analytical workloads. Since Spark 2.0 release, the new Tungsten execution engine in Apache Spark implements whole-stage code generation, a technique inspired by modern compilers to collapse the entire query into a single function. This JIT compiler approach is a far superior architecture than the row-at-a-time processing or code generation model employed by other engines, making Spark one of the most efficient in the market.

For the past few months, we have been busy working on the next major release of the big data open source software we love: Apache Spark 2.0. Since Spark 1.0 came out two years ago, we have heard praises and complaints. Spark 2.0 builds on what we have learned in the past two years, doubling down on what users love and improving on what users lament. While this blog summarizes the three major thrusts and themes--easier, faster, and smarter--that comprise Spark 2.0, the themes highlighted here deserve deep-dive discussions that we will follow up with in-depth blogs in the next few weeks. Before we dive in, we are happy to announce the availability of the Apache Spark 2.0 technical preview in Databricks Community Edition today.